System and method for enabling mirror video chat using a wearable display device

ABSTRACT

A method of exchanging audio-visual communication information between users includes detecting using an image capturing device associated with a wearable communication device, a mirror or image reflecting surface disposed in an environment of a first user, detecting a boundary of the mirror or image reflecting surface in response to the mirror being detected in the environment of the first user, selecting a portion of a first image displayed on the mirror or image reflecting surface within the boundary of the mirror or image reflecting surface, and displaying the portion of the first image as an overlay on a second image to a second user. A corresponding system and computer-readable device are also disclosed.

BACKGROUND

Technical Field

The present disclosure relates to a system and method for enabling amirror video chat session in which audio-visual information is exchangedbetween users, and more specifically relates to a system and method forimplementing a wearable computing device or similar communication deviceduring a video chat session by detecting and using a mirror or imagereflecting surface located in the user's surrounding environment so thatthe user's own image can be exchanged with remote participants.

Related Art

Wearable computing devices in the form of glasses or similarhead-mounted face apparatus or interactive head-mounted eye-pieces aregenerally wearable computers with an optical head-mounted display (OHMD)with an integrated processor and an integrated image source fordelivering content to an optical assembly. The user is able to viewvarious information and images regarding the surrounding environment insmartphone-style hands-free format. These devices can also communicatevia the Internet using natural language voice commands.

These wearable devices have the ability to take photos of theenvironment, record high-definition quality views of the surroundingenvironment, while simultaneously keeping the viewing screen on. Theuser is able to control the functions of the wearable devices usingtouchpad controls, which are built into the side of the device near thestem of the wearable display or similar electronic housing locatedotherwise in the frame portion of the device near the stem or extendedaway from the stem. The user controls the device by swiping the touchpadthrough an interactive interface displayed on the screen the user issimultaneously viewing. For example, a user can slide the touchpad in acertain direction to display current events, news, weather, whilesliding the touchpad in a different direction can generate a series ofpast events such as photos, received phone calls, photos, and otherkinds of information.

SUMMARY

Embodiments of the disclosure will become apparent from the followingdetailed description considered in conjunction with the accompanyingdrawings. It is to be understood, however, that the drawings aredesigned as an illustration only and not as a definition of the limitsof this disclosure.

The disclosed technology is directed to a method of exchangingaudio-visual communication information between users, which includesdetecting an image reflecting surface disposed in an environment of afirst user, detecting a boundary of the image reflecting surface inresponse to the image reflecting surface being detected in theenvironment of the first user, selecting a portion of a first imagedisplayed on the image reflecting surface within the boundary of theimage reflecting surface and displaying the portion of the first imageas an overlay on a second image to a second user.

The method of exchanging audio-visual communication information mayfurther include controlling selection of the portion of the first imagein response to an image defining gesture associated with the first user.The method may include modifying a size of the portion of the firstimage in response to the image defining gesture associated with thefirst user. The method may also include displaying the portion of thefirst image as an overlay on the second image on the image reflectingsurface disposed in an environment of the first user that can be viewedby the first user. Rotating the image capturing device may occur inresponse to the first user. Rotating the image capturing device mayoccur in response to a sensory image signal detected by the imagecapturing device. Generating a tactile signal to the wearablecommunication device associated with the first user may occur.Generating a tactile signal to the wearable communication deviceassociated with the first user may occur in response to a sensorydetection signal associated with movement by the first user towards anedge of the image reflecting surface. Generating a tactile signal to thewearable communication device associated with the first user may resultin response to a sensory detection signal associated with movement bythe first user outside the boundary of the image reflecting surface.

The disclosed technology is further directed to an apparatus to exchangeaudio-visual communication information between a first user of awearable communication device and a second user which includes an imagecapturing device, a processing device, and a memory to storeinstructions that, when executed by the processing device, performoperations comprising detecting an image reflecting surface disposed inan environment of a first user, detecting a boundary of the imagereflecting surface in response to the image reflecting surface beingdetected in the environment of the first user, selecting a portion of afirst image displayed on the image reflecting surface within theboundary of the image reflecting surface and displaying the portion ofthe first image as an overlay on a second image to a second user.

The disclosed technology is yet further directed to a computer-readabledevice to store instructions that, when executed by a processing device,perform operations to exchange audio-visual communication informationbetween users, wherein the operations include detecting an imagereflecting surface disposed in an environment of a first user, detectinga boundary of the image reflecting surface in response to the imagereflecting surface being detected in the environment of the first user,selecting a portion of a first image displayed on the image reflectingsurface within the boundary of the image reflecting surface, anddisplaying a portion of the first image as an overlay on a second imageto a second user.

The disclosed technology is yet further directed to a system and methodof engaging one or more third parties in a video chat session or similarvideo conference session using a mirror or image reflecting surface thatis detected in the environment, while also permitting the participantsto view the user, as well as leverage the view provided in the mirror orimage reflecting surface for chatting with same remote participants. Thesystem and method of engaging in a mirror video chat with at least onethird-party participant includes using a head-mounted display having afront-facing camera, an audio input and output, preferably microphoneand speakers that are embedded in the head-mounted display, a wirelessconnection to a computing unit (such as a mobile phone or laptop) thatcan process the video and audio information captured by the front-facingcamera or similar image capturing device.

The disclosed technology is yet further directed to a system and methodusing a processing device, such as in a wearable computer display thatrecognizes a user's gesturing signal. Such hand or other kind ofgesturing permits the user to delineate which portion of his or hercaptured video image will be delivered to the third-party remoteparticipants during the video chat session. The user may use any numberof hand gestures, such as by expanding or contracting a thumb andforefinger, or moving a pointing finger upwards or downwards to capturea desired portion of the user's own video image.

The disclosed technology is yet further directed to using a mirror orimage reflecting surface as a visual screen for viewing the capturedvideo image of the user as superimposed in a coherent fashion on thethird-party participant's incoming video feed. The user may view suchsuperimposed video feeds either through the head-mounted wearabledisplay or using the mirror or image reflecting surface in the user'senvironment as a viewing screen for the superimposed video image.

Embodiments will become apparent from the following detaileddescription, which is to be read in connection with the accompanyingdrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings constitute a part of this disclosure and include examples,which may be implemented in various forms. It is to be understood thatin some instances, various aspects of the disclosure may be shownexaggerated or enlarged to facilitate understanding. The teaching of thedisclosure can be readily understood by considering the followingdetailed description in conjunction with the accompanying drawings.

FIG. 1 is an illustration of a first embodiment of wearable displayglasses.

FIG. 2 is an illustration of a second embodiment of wearable displayglasses.

FIG. 3 is a flowchart that illustrates an example method of a mirrorvideo chat session using a wearable display and mirror.

FIG. 4 is a flow chart illustrating a de-warping process as applied tocaptured video images.

FIG. 5 is a block diagram showing an exemplary embodiment of a computernetwork infrastructure.

FIG. 6 is a block diagram showing a portion of an exemplary machine inthe form of a computing system configured to perform methods accordingto one or more embodiments.

It is to be appreciated that elements in the figures are illustrated forsimplicity and clarity. Common but well-understood elements, which maybe useful or necessary in a commercially feasible embodiment, are notnecessarily shown in order to facilitate a less hindered view of theillustrated embodiments.

DETAILED DESCRIPTION

While wearable display devices, such as GOOGLE® glasses, are considereda great solution for displaying information in the environment, thesewearable devices or eyepieces only provide the users with a front-facingcamera that makes it impossible to share the wearer's own face or imageduring a video chat session, videoconference session or any otherstandard chat software sessions, such as Skype®. Thus, there is a needfor a wearable display device that permits the user of the displaydevice to project their own image to participants of the chat or videoconference session. In some embodiments, the user is also permitted toview the remote participants as an overlay image on the mirror or imagereflecting surface when viewing the mirror or image reflecting surface.

The disclosed system and method detects mirrors in the environment, thusallowing the participants to leverage the view provided in the mirror orimage reflecting surface for chatting with remote third-partyparticipants and for various other purposes, described in greater detailbelow. One of the greater concerns with portable electronic devices isthat they detract from in-person social element of interacting withhumans. While disclosed embodiments are not actually in-personexperiences, it is believed that they permit as close to an in-personexperience as possible while using a wearable display, especially in aneducational or professional setting while generally avoiding the expenserequired to set-up other video conferencing systems.

There is also a need for various applications of the disclosedembodiments, for example in the health field. The disclosed system andmethod would permit a team of doctors to remotely consult about apatient's case. The front-facing camera would permit the doctor to showone or more patients' x-ray images while also stepping in front of themirror and permitting the doctor to begin a face-to-face conference witha remote team of participants. This smooth transition between focusingon the task and speaking with the team is more natural, and closer tothe aura of an in-person interaction than mere standard video chatapplications.

There is a further need for an application in which the mirror chatsession would allow students that are home-schooled or live in remoteareas to attend a virtual classroom with a teacher. They would be ableto share both their work (using the front-facing camera) while alsopermitting face-to-face conversations with the teacher and otherstudents participating in the session using the disclosed embodiments.

Yet another instance is the application in a remote work setting. Acompany can implement inexpensive mirror or image reflecting surface toachieve the same effect as more expensive systems, such as Cisco®telepresence systems. Essentially, the conventional setup of monitorswould be replaced with mirrors of the same shape and size. If each ofthe participants meets the standard requirements, the session can beimplemented in the same fashion as the Cisco® telepresence system orsimilar videoconferencing systems.

Embodiments disclosed herein perform a mirror video chat session with aremote third party participant using a wearable display device includinga front-facing camera and a mirror or image reflecting surface detectedin the user's surrounding environment.

Wearable computing devices, as shown in FIGS. 1 and 2, include devicessuch as GOOGLE® glasses. These head-mounted devices (20) may includelens elements (21) and (26) that may be sufficiently transparent toallow users to see through the lens elements. They may also be formed ofany materials that can display a graphic image or other graphicrepresentation. Thereby, the wearable display devices may permit analtered display where the projected image or graphic is superimposedover a real-world view as perceived by the user through the lenselements.

The wearable devices may include an on-board computing system (24) whichmay be positioned on the extending side-arm of the device (24), a videocamera (22) or similar image capturing device, one or more sensor(s)located on the side frame portion of the device and one or morefinger-operable touch pad (23) or similar interactive device. Thecomputing system may be positioned on other parts of the head-mounteddevice or positioned remotely from the head-mounted device, andconnected to the head-mounted device (20) wirelessly or through a wiredconnection. The computing device may also be configured to receive andanalyze data from the video camera (22) and the finger-operable touchpad (23) and, in other embodiments, from other sensory devices, userinterfaces or both, and generate images for output by the lens elements(21) or (26). The computing device may be programmed to recognize handgesturing motions via sensors. Such hand gesturing motions are meant todelineate, by sending appropriate control signals to the computingdevice, which portion of the captured video image of the user is to besent to the remote third-party participants. The video camera (22) orsimilar image capturing device may be configured to capture images atvarious resolutions and at different frame rates.

As shown in FIG. 3, the user initiates a mirror video chat session witha third-party while simultaneously wearing the computing device, such asGOOGLE® glasses or similar wearable display device, as shown in FIGS. 1and 2. The wearable display that would be implemented during the mirrorvideo chat session may include an infrared component or infrared typelight, a camera or alternatively, an image capturing device and one ormore processing device(s). In accordance with the embodiment shown inFIG. 3, the user initiates a mirror video chat session (2) using videochat software, such as, Skype®, Google Hangouts, or alternative chat orvideoconferencing software. Since mirror video chat is a technique thatessentially spoofs a camera using a mirror or image reflecting surface,it can be transparently applied to any existing video chat software. Theuser may initiate the mirror video chat session using controls availablein the wearable device. In certain embodiments, the user may use voicecontrol commands such as “Skype® name of person” or select a command onan operable touch pad to initiate the mirror chat session.

The third-party remote participants are not required to use the mirrorchat video system unless they are also using a wearable display devicewith a front-facing camera. The remote participants can implement eitherthe disclosed mirror chat system or may implement a standard video chatsystem, which typically includes a webcam, speakers, microphone, andnetwork connectivity, with the remote video chat participants.

The system will next generate a control signal to detect whether amirror or image reflecting surface of any size is present in theenvironment as shown in step (3). The system uses the front-facingcamera in the wearable display device to scan for and detect thepresence of a mirror or image reflecting surface in the user'ssurrounding environment. In certain embodiments, the system activatesthe front facing camera and asks the user to turn his or her head fromside-to-side, analyzing the image feed from the camera. All objects thatare reflected as images from the mirror shift positions in a way thatdiffers from how objects in non-mirrored or non-image reflectingsurfaces appear to a camera. This change in orientation of imagesreflected from the mirror or image reflecting surface and captured bythe camera essentially allows the edges of the mirror or imagereflecting surface to be identified and such detection can beimplemented using a computer vision library application such as OpenCV.

The system includes an image capturing device with optical systems thatwill continuously capture images in the surrounding environment whilesimultaneously running system check(s) to detect for the gradient colorand surface brightness of mirrors or image reflecting surfaces (with theexception of colored or antique mirrors) that generally match a standardgradient color and surface brightness of the mirror. The processor willdetermine whether the captured video image matches the template storedin the memory of the device for gradient color and surface brightness ofmirrors or image reflecting surfaces.

A mirror generally has no color unless it is a mirror of a colored kindother than a standard silver mirror. An ideal mirror will reflect alllight equally and spectrally. A mirror generates a spectral reflectionin which all light hitting the mirror from a single direction isreflected in a single direction as it leaves the surface. Mirrors aregenerally made behind a layer of glass. The glass itself can sometimesabsorb certain colors and so the mirror does not reflect all colorsfully. In this case, it may appear that the mirror has a color.

Ideally a silver mirror, which is a generally a polished silver platedsurface covering a layer of plain glass and generally reflectingwhatever color is applied to its surface, would be present in theenvironment and can be detected by the mirror detection device. However,the image detection device can also be configured to scan for any imageand/or any type of mirror (whether colored or silver).

The system will continuously scan to detect the presence of a mirror orimage reflecting surface including searching for a reflection of theuser if the system is programmed to search for the user's own image. Thesystem may also scan to detect reflection of the actual mirror detectiondevice or the housing in which it resides if programmed to detect suchimages as a first sub-step of actual mirror detection.

The mirror or image reflecting surface detection device or similarprocessing device may also be configured to detect for the image of theactual mirror detection device or the housing in which the mirrordetection device resides in one embodiment. The device may bepre-programmed to detect the reflection of such image(s) from a mirroror image reflecting surface (which are stored in a memory device priorto the video chat session or prior to the mirror detection step (3)). Astep of comparing currently captured images to images stored in thememory device is performed by the mirror detection device and, once amatch is detected, the mirror detection device generates a controlsignal that indicates that a mirror or similar image reflecting surfacehas been detected, and the system proceeds to the mirror edge detectionstep (4).

The system next scans, via a mirror edge detection device or processingdevice, for the boundaries of the already detected mirror or imagereflecting surface through the optical properties of the reflectedimage. A shown in step 4 of FIG. 1, once the system has generated, asignal indicating that the mirror or image reflecting surface wasdetected in the surrounding environment of the wearable display user,the system will next generate a signal, via the detection device, toscan for and detect the edges of the mirrors. Using the detectedgradient color and surface optics of the already detected mirror, thedetection unit will scan for and detect the entire surface area of themirror until an outermost boundary perimeter of the mirror or imagereflecting surface is detected, and thus delineated by the detectiondevice as location data. The outer boundary parameters are then used bythe detection device to process the image that is being reflected offthe mirror.

In some embodiments, the user may also assist the detection device orprocessing device by motioning towards the boundary edges so that thesystem detects the user's manual selection of the mirror's boundariesfrom a visual or other sensory cue detected from the user's motion. Ifthe mirror is a compact size mirror, the user's head motion will cue thedetection device as to the boundaries of the mirror. If the mirror orimage reflecting surface is a larger size, the user can motion byshifting sideways left or right from the center of the image reflectingsurface towards the outer edge to direct the detection device as to thelocation of the outermost boundaries. There are other possibleembodiments in which the detection device detects the edges of themirror or image reflecting surface.

Once the mirror chat session is initiated and a mirror or imagereflecting surface in the environment is detected, the system mayrequest that the user select the area of the mirror or image reflectingsurface to share with the third party participant. The user may useimage defining gesturing to delineate and/or further define the area ofthe mirror the user wants to share with the third party participant.

Once a mirror is detected by the mirror detection device of the system,the system, via the front-facing camera or other image capturing device,next captures the displayed image being reflected from the mirror orimage reflecting surface, which in this embodiment is the user's ownimage, whether a full-sized image of the user's body, a partial portionof the user's image, a portion of the facial image or the full facialimage. In certain embodiments, the system may also capture multipleimages reflected from the mirror or image reflecting surface thatinclude the user's own image.

The video frame is next scaled as shown in step (5) by image processingtechniques through one- or two-dimensional linear image scaling unit(s)and/or similar processes using devices for accomplishing either themagnification or reduction of the image by scaling pixel or image dataincluding techniques that include sharpness enhancement and transientimprovement in either or both of the horizontal and vertical dimensionsof the image data. Some image scaling and sharpness enhancementstechniques by way of example are disclosed in U.S. Pat. No. 7,782,401,which is incorporated herein by reference.

The eye detection step is next initiated as shown in step (6) of FIG. 3.The system will scan the entire image that is currently captured by thedetection device or image capturing device. Generally, while theindividual's own image is simultaneously viewed on the mirror or imagereflecting surface, the eye detection process is commenced throughprocesses in the art. Eye detection is accomplished using techniquessuch as OpenCV Haarlike eye-detection algorithm. Once the eyes aredetected, the system measures the distance between the eyes as ameasuring unit and will suggest an initial sharing region of 4 unitswide and 5 units tall centered around the detected and measured eyearea. Essentially, even without the user specifying a region to share,this permits the system to intelligently estimate which area of themirror or image reflecting surface includes the user's face.Additionally, continual detection of the eyes, allows the proper regionof the mirror or image reflecting surface to be captured by the cameraand shared with the third-party participants as a continuous video feedeven should the user slightly shift his or her position while in frontof the mirror or image reflecting surface.

By way of example, methods to scan for irises include an automatedmethod of biometric identification that uses mathematicalpattern-recognition technology on video images, specifically on theirises of a person's eyes, which consist of complex random patterns thatare unique which by way of example is disclosed in U.S. Pat. No.8,489,887, the disclosure of which is incorporated by reference. Inaddition, as disclosed in U.S. Pat. No. 8,457,363, which is incorporatedby reference herein, another technique for eye detection in an imageincludes accurately detecting eyes by dividing an input face image intoleft and right images, detecting eye candidates from limited imageregions of the left and right images, evaluating the detected eyecandidates using geometric information, and verifying the eye candidatesusing a support vector machine (SVM) algorithm.

Another technique includes retinal scanning, which uses cameratechnology with subtle infrared illumination to acquire images of thedetails of the iris. Another technique is disclosed by way of example inU.S. Pat. No. 8,090,157, the disclosure of which is incorporated byreference herein. The '157 patent describes a technique, whereby the eyeis illuminated to generate a reflection reference point on the pupilsurface. The captured wide-field image is filtered using reflectiondetection contrast changes to find potential eye locations. For eachpotential eye location, the local contrast change between the centralpoint and its surrounding pixels is computed and results are prioritizedto extract valid eye pair with maximum local contrast change. The eyepair is defined as a valid pair if the two potential eyes are spacedwithin a predefined range. The following techniques are, by way ofexample, and the current disclosure is not limited to any particulartechnique for eye detection processes.

Once the system has detected the eye(s) or the iris portion of theeye(s) of the person whose image is currently projected on the mirror orimage reflecting surface in the environment, the system next crops thecaptured video feed as shown in step (7) through cropping tools, such asComputer Vision, Open Source®. Cropping is used to eliminate unwantedportions of the video image. The eye detection data will generate, instep (6,) via a processing device, an already mapped location of theuser's eyes. The system may use this eye detection data to center theuser's image through image processing and cropping techniques togenerate the final video feed that will be delivered to the remote usersor third-party video chat participants. Libraries available in OpenCVcan also be used to implement cropping techniques during step (7) of themirror chat process.

Once the video feed is cropped, the system will next run a check of theimage to determine whether the user defines the video space (8) asdefined by the boundaries of the currently cropped video feed. Theuser's image is compared against the currently cropped video feed and,if the user is not centered or essentially defining the video space, thesystem will generate a signal to continuously crop the image as shown insteps (7) and (8) of FIG. 3 until the user defines the video space.

The disclosed system next de-warps (9) the captured images for anydistortion by processing the image through de-warping process (40) suchas that shown in FIG. 4. In this process as described in U.S. Pat. No.8,068,695, the user identifies representative features (41) of theobject, determines vectors characterizing respective shapes or positionsof portions of the identified features (42), selects portions of thedetermined vectors that have valid data at corresponding positions andapplies (44) principal component analysis to the selected vectorportions. The output of the principal component analysis includes atleast one basis vector and at least one corresponding coefficient withrespective sets of at least one coefficient corresponding to each of theselected vector portions. The principal component analysis alsogenerates an average vector describing an average of the selectedvectors (45). Together, the average vector, at least one basis vectorand sets of coefficients provide an estimate of the curvature ordistortion of the object at positions in the image represented by theselected vector portions. The method includesinterpolating/extrapolating additional and optionally alternativedistortion estimates based on output from the application (44). Theoutput or de-warp map generated by this interpolation/extrapolation (45)can then be used to de-warp (46) at least a portion of the image or afirst area associated with the selected vector portions (43). Thede-warped image or image portion can then be stored (47) or used torender (48) a corrected or compensated version of the image.

The disclosed system next will proceed to use the captured image to jointhe user into the video chat session or the image may also be flipped tocorrect for the image reversal caused by the reflection of the user'simage from the mirror or image reflecting surface. The user may nowenter into a video-conference with the remote third-party video chatparticipants. The user can now view live video images of the third-partyparticipants through the eyepiece included in the wearable eye display.

The mirror itself may also now act as a viewable screen onto which theuser's viewable images in the wearable display device will besuperimposed with the live video feed of the remote third-partyparticipants and simultaneously projected onto the mirror or imagereflecting surface during the mirror video chat session. The wearabledisplay user can now view the mirror or image reflecting surface toparticipate and engage the remote third-party participants in the mirrorvideo chat session while also viewing his or her own imagesimultaneously.

In certain embodiments, the user may also select portions of the imagethey want to share with third-party participants. The system may includeair motion detection sensors or similar sensor(s) which may detect auser's hand gestures to capture a certain portion of the projected imagereflected off the mirror or image reflecting surface. The user may usetheir thumb and forefinger for example and initiate the gesture with theuser's thumb and forefinger touching which is also visible through thecamera's eyepiece to the user and currently centered on the portion ofthe image of interest. The user may then expand open the thumb andforefinger towards the outer boundaries of the portion of their ownimage that they intend to share with the remote third-party videoparticipants. The system may also be configured to detect other finger,hand waving motions, or other gestures that essentially define and framethe portion of the user's video image that they would like to exchangewith the remote third-party participants. The control device of thesystem processor may be configured to generate a control command toframe a portion of the user's video image in response to a pointinggesture or other gesture associated with the displayed processorcontent.

The disclosed system may also be configured to provide tactile or haptictype feedback such as generating a vibrate or similar resistance typesignal which is delivered to the side of the wearable device or in whichthe user may have moved out of the focal point of the currentlyprojected image to remote third-party participants. In addition, ahaptic type signal may be generated when a user lean's outside the focalpoint of the currently projected image to remote third-partyparticipants. As an example, as the user approached the right edge ofthe mirror or image reflecting surface, the user may sense a slightvibration which is emitted from the right side of the wearable device.As the user leans closer to the edge of the mirror or image reflectingsurface, actually stepping outside the line of view of the mirror ascaptured by the camera, the vibration may intensify to signal to theuser of the wearable device that he or she has left the entire area ofpossible image capture for the camera. Essentially this feature would bea signal to the user that translates to the side of the mirror in whichthe user has stepped very close or closer to the edge of the mirror. Thesignal would strengthen if the user has completely stepped out of themirror's line of view and essentially, the scope of the camera's imagecapturing area. The signal may also serve as a tactile guide to the userto step more towards the center of the mirror or image reflectingsurface and more towards the opposite side of the user's wearable devicein which the signal is received or within the scope of the camera'simage capturing area.

In yet another embodiment, a process is performed for the removal ofbright spots which may be reflected from the mirror or image reflectingsurface and which may impede viewing of the video image of the user bythe remote third-party participants and for the user as well. The devicemay detect bright spots on the captured image, merge the image andremove these bright spots through de-warping processes. This feature mayalso be accomplished in some embodiments using libraries in OpenCV thataccomplish bright spot removal. The system may implement OpenCV tolocate an area in the image that appears brighter than the averagebrightness of the overall image. A filter is then implemented to reducethe areas having higher levels of brightness which distort the videofeed.

In some embodiments, the disclosed system using an image processingdevice, operates to process image data such as detected bright spot(s)or high-contrast area(s) in the captured video image(s), morespecifically to images reflected from a mirror-like surface. An imagede-warper de-warps image data according to obtained sets of positionestimates and generates a compensated version of the image data withunwanted anomalies such as the detected bright spots or observed high“contrast areas”. At least one of an image data storage device isoperative to store the compensated version of the image data and arendering device is operative to render the compensated version of theimage data. At least one of the feature identifier, the feature vectordeterminer, the vector selector, the principal component analyzer, thecorrection estimator, the bright-spot image de-warper, the image datastorage device and the rendering device is implemented in hardware or acombination of hardware and/or software to generate a compensatedversion of the image data through such de-warping techniques.

FIG. 5 is an example schematic diagram of a computer networkinfrastructure implemented in the disclosed mirror chat system. Thesystem (50) includes a device (51) that communicates using acommunications link (56) (e.g., wired or wireless) to a remote device(55) and/or to an iCloud network and/or iCloud network driver (58). Thedevice may be any type of device configured to receive data and displayinformation corresponding to the exchange of audio-visual data betweenusers. The device may be a head-mounted wearable display device (20) orsimilar wearable computing device as shown in FIG. 1. The device (51)may include a display system (57) comprising a processor (52) and adisplay (54). The processor (52) may receive data from the remote device(55) and configure the data for display on the display (54). The displaymay also be projected onto a mirror (59) and the user's own image (60)may be visible to the remote participants via the processing andexchange of data between the device (51) and one or more remotedevice(s) (55). The device (51) may also include on-board data storage,such as a memory (53) coupled to processor (52) such as a microprocessoror a digital signal processor. The memory (53) may store software andother data that can be retrieved and executed by the processor (52), forexample. The remote device (55) may be any type of computing device ortransmitter including a laptop computer, a mobile telephone or tablecomputing device, etc. that is configured to transmit data to the device(51). The remote device (55) and the device (51) may contain hardware toenable the one or more communication link(s) (56), via processors,transmitters, receivers, antennas, etc. between the wearable computingdevice (51), one or more remote device(s) (55) and an iCloud (58).

As shown in FIG. 5, the communication link(s) (56) is preferably awireless connection. However, wired connections may also be implementedsuch as a wired serial bus or a parallel or other proprietaryconnection. The communication link(s) (56) may be implemented wirelesslyvia a wireless connection such as Bluetooth® radio technology, cellulartechnologies such as GSM, CDMA, WiMAX, LTE®, communication protocolssuch as IEEE 802.11 including revisions thereto, Zigbee® technology,among other wireless configurations. The remote device(s) (55) may beaccessible via the Internet and may include a computer clusterassociated with a particular web service such as Skype®, FaceTime,social networking, photo sharing, and the like.

There are a number of advantages of using mirror chat rather than knownand more complex videoconference systems in that it is much less costlyto purchase the special equipment required for most videoconferenceapparatus. Mirror chat requires no special equipment other than a mirroror image reflecting surface in the environment and a wearable displaydevice or similar computing device with forward-facing camera or similarimage capturing device.

An additional significant advantage in using mirror chat is that eyegaze correction can be accomplished with greater precision and withoutany additional significant processing. This is essentially a naturalresult of using a mirror or image reflecting surface instead of a camerato capture one's own image. It is a known problem, that when one openstheir webcam view on their computers or a camera from their smartphoneand targets their own image, it appears as if the image does not lookexactly as the image being captured. A property of using a camera tocapture one's own image is that you are generally unable to make eyecontact naturally without some distortion (for example, eye gazedistortion) when actually making eye contact with the computer or thesmartphone. However, when using the mirror or image reflecting surface,your image appears exactly as reflected from the mirror or imagereflecting surface without processing the user's eye gaze such that amore natural eye gaze is captured rather than the eye gaze distortiontypically experienced with computer or smartphone cameras and/or similarkinds of cameras implemented in processing devices.

FIG. 6 is a block diagram of an embodiment of a machine in the form of acomputing system 100, within which a set of instructions 102, that whenexecuted, may cause the machine to perform any one or more of themethodologies disclosed herein. In some embodiments, the machineoperates as a standalone device. In some embodiments, the machine may beconnected (e.g., using a network) to other machines. In a networkedimplementation, the machine may operate in the capacity of a server or aclient user machine in a server-client user network environment. Themachine may comprise a server computer, a client user computer, apersonal computer (PC), a tablet PC, a personal digital assistant (PDA),a cellular telephone, a mobile device, a palmtop computer, a laptopcomputer, a desktop computer, a communication device, a personal trusteddevice, a web appliance, a network router, a switch or bridge, or anymachine capable of executing a set of instructions (sequential orotherwise) that specify actions to be taken by that machine.

The computing system 100 may include a processing device(s) 104 (e.g., acentral processing unit (CPU), a graphics processing unit (GPU), orboth), program memory device(s) 106, and data memory device(s) 108,which communicate with each other via a bus 110. The computing system100 may further include display device(s) 112 (e.g., liquid crystalsdisplay (LCD), a flat panel, a solid state display, or a cathode raytube (CRT)). The computing system 100 may include input device(s) 146(e.g., a keyboard), cursor control device(s) 116 (e.g., a mouse), diskdrive unit(s) 118, signal generation device(s) 119 (e.g., a speaker orremote control), and network interface device(s) 124.

The disk drive unit(s) 118 may include machine-readable medium(s) 120,on which is stored one or more sets of instructions 102 (e.g., software)embodying any one or more of the methodologies or functions disclosedherein, including those methods illustrated herein. The instructions 81may also reside, completely or at least partially, within the programmemory device(s) 106, the data memory device(s) 108, and/or within theprocessing device(s) 104 during execution thereof by the computingsystem 100. The program memory device(s) 106 and the processingdevice(s) 104 may also constitute machine-readable media. Dedicatedhardware implementations, not limited to application specific integratedcircuits, programmable logic arrays, and other hardware devices canlikewise be constructed to implement the methods described herein.Applications that may include the apparatus and systems of variousembodiments broadly include a variety of electronic and computersystems. Some embodiments implement functions in two or more specificinterconnected hardware modules or devices with related control and datasignals communicated between and through the modules, or as portions ofan application-specific integrated circuit. Thus, the example system isapplicable to software, firmware, and hardware implementations.

In accordance with various embodiments of the present disclosure, themethods described herein are intended for operation as software programsrunning on a computer processor. Furthermore, software implementationscan include, but not limited to, distributed processing orcomponent/object distributed processing, parallel processing, or virtualmachine processing can also be constructed to implement the methodsdescribed herein.

The present embodiment contemplates a machine-readable medium orcomputer-readable medium containing instructions 102, or that whichreceives and executes instructions 102 from a propagated signal so thata device connected to a network environment 122 can send or receivevoice, video or data, and to communicate over the network 122 using theinstructions 102. The instructions 102 may further be transmitted orreceived over a network 122 via the network interface device(s) 124. Themachine-readable medium may also contain a data structure for storingdata useful in providing a functional relationship between the data anda machine or computer in an illustrative embodiment of the disclosedsystems and methods.

While the machine-readable medium 120 is shown in an example embodimentto be a single medium, the term “machine-readable medium” should betaken to include a single medium or multiple media (e.g., a centralizedor distributed database, and/or associated caches and servers) thatstore the one or more sets of instructions. The term “machine-readablemedium” shall also be taken to include any medium that is capable ofstoring, encoding, or carrying a set of instructions for execution bythe machine and that cause the machine to perform anyone or more of themethodologies of the present embodiment. The term “machine-readablemedium” shall accordingly be taken to include, but not be limited to:solid-state memories such as a memory card or other package that housesone or more read-only (non-volatile) memories, random access memories,or other re-writable (volatile) memories; magneto-optical or opticalmedium such as a disk or tape; and/or a digital file attachment toe-mail or other self-contained information archive or set of archives isconsidered a distribution medium equivalent to a tangible storagemedium. Accordingly, the embodiment is considered to include anyone ormore of a tangible machine-readable medium or a tangible distributionmedium, as listed herein and including art-recognized equivalents andsuccessor media, in which the software implementations herein arestored.

Although the present specification describes components and functionsimplemented in the embodiments with reference to particular standardsand protocols, the disclosed embodiment are not limited to suchstandards and protocols.

The illustrations of embodiments described herein are intended toprovide a general understanding of the structure of various embodiments,and they are not intended to serve as a complete description of all theelements and features of apparatus and systems that might make use ofthe structures described herein. Many other embodiments will be apparentto those of skill in the art upon reviewing the above description. Otherembodiments may be utilized and derived there from, such that structuraland logical substitutions and changes may be made without departing fromthe scope of this disclosure. Figures are also merely representationaland may not be drawn to scale. Certain proportions thereof may beexaggerated, while others may be minimized. Accordingly, thespecification and drawings are to be regarded in an illustrative ratherthan a restrictive sense.

Such embodiments of the inventive subject matter may be referred toherein, individually and/or collectively, by the term “embodiment”merely for convenience and without intending to voluntarily limit thescope of this application to any single embodiment or inventive conceptif more than one is in fact disclosed. Thus, although specificembodiments have been illustrated and described herein, it should beappreciated that any arrangement calculated to achieve the same purposemay be substituted for the specific embodiments shown. This disclosureis intended to cover any and all adaptations or variations of variousembodiments. Combinations of the above embodiments, and otherembodiments not specifically described herein, will be apparent to thoseof skill in the art upon reviewing the above description.

The Abstract is provided to comply with 31 C.F.R. §1.12(b), whichrequires an abstract that will allow the reader to quickly ascertain thenature of the technical disclosure. It is submitted with theunderstanding that it will not be used to interpret or limit the scopeor meaning of the claims. In addition, in the foregoing DetailedDescription, it can be seen that various features are grouped togetherin a single embodiment for the purpose of streamlining the disclosure.This method of disclosure is not to be interpreted as reflecting anintention that the claimed embodiments require more features than areexpressly recited in each claim. Rather, as the following claimsreflect, inventive subject matter lies in less than all features of asingle disclosed embodiment. Thus the following claims are herebyincorporated into the Detailed Description, with each claim standing onits own as a separately claimed subject matter.

In a particular non-limiting, example embodiment, the computer-readablemedium can include a solid-state memory such as a memory card or otherpackage that houses one or more non-volatile read-only memories.Further, the computer-readable medium can be a random access memory orother volatile re-writable memory. Additionally, the computer-readablemedium can include a magneto-optical or optical medium, such as a diskor tapes or other storage device to capture carrier wave signals such asa signal communicated over a transmission medium. A digital fileattachment to an e-mail or other self-contained information archive orset of archives may be considered a distribution medium that isequivalent to a tangible storage medium. Accordingly, the disclosure isconsidered to include any one or more of a computer-readable medium or adistribution medium and other equivalents and successor media, in whichdata or instructions may be stored.

In accordance with various embodiments, the methods, functions or logicdescribed herein may be implemented as one or more software programsrunning on a computer processor. Dedicated hardware implementationsincluding, but not limited to, application specific integrated circuits,programmable logic arrays and other hardware devices can likewise beconstructed to implement the methods described herein. Furthermore,alternative software implementations including, but not limited to,distributed processing or component/object distributed processing,parallel processing, or virtual machine processing can also beconstructed to implement the methods, functions or logic describedherein.

It should also be noted that software which implements the disclosedmethods, functions or logic may optionally be stored on a tangiblestorage medium, such as: a magnetic medium, such as a disk or tape; amagneto-optical or optical medium, such as a disk; or a solid statemedium, such as a memory card or other package that houses one or moreread-only (non-volatile) memories, random access memories, or otherre-writable (volatile) memories. A digital file attachment to e-mail orother self-contained information archive or set of archives isconsidered a distribution medium equivalent to a tangible storagemedium. Accordingly, the disclosure is considered to include a tangiblestorage medium or distribution medium as listed herein, and otherequivalents and successor media, in which the software implementationsherein may be stored.

Although specific example embodiments have been described, it will beevident that various modifications and changes may be made to theseembodiments without departing from the broader scope of the inventivesubject matter described herein. Accordingly, the specification anddrawings are to be regarded in an illustrative rather than a restrictivesense. The accompanying drawings that form a part hereof, show by way ofillustration, and not of limitation, specific embodiments in which thesubject matter may be practiced. The embodiments illustrated aredescribed in sufficient detail to enable those skilled in the art topractice the teachings disclosed herein. Other embodiments may beutilized and derived therefrom, such that structural and logicalsubstitutions and changes may be made without departing from the scopeof this disclosure. This Detailed Description, therefore, is not to betaken in a limiting sense, and the scope of various embodiments isdefined only by the appended claims, along with the full range ofequivalents to which such claims are entitled.

Such embodiments of the inventive subject matter may be referred toherein, individually and/or collectively, by the term “embodiment”merely for convenience and without intending to voluntarily limit thescope of this application to any single embodiment or inventive conceptif more than one is in fact disclosed. Thus, although specificembodiments have been illustrated and described herein, it should beappreciated that any arrangement calculated to achieve the same purposemay be substituted for the specific embodiments shown. This disclosureis intended to cover any and all adaptations or variations of variousembodiments. Combinations of the above embodiments, and otherembodiments not specifically described herein, will be apparent to thoseof skill in the art upon reviewing the above description.

In the foregoing description of the embodiments, various features aregrouped together in a single embodiment for the purpose of streamliningthe disclosure. This method of disclosure is not to be interpreted asreflecting that the claimed embodiments have more features than areexpressly recited in each claim. Rather, as the following claimsreflect, inventive subject matter lies in less than all features of asingle disclosed embodiment. Thus the following claims are herebyincorporated into the Detailed Description, with each claim standing onits own as a separate example embodiment.

Although preferred embodiments have been described herein with referenceto the accompanying drawings, it is to be understood that the disclosureis not limited to those precise embodiments and that various otherchanges and modifications may be affected herein by one skilled in theart without departing from the scope or spirit of the embodiments, andthat it is intended to claim all such changes and modifications thatfall within the scope of this disclosure.

What is claimed is:
 1. A method of exchanging audio-visual communicationinformation between users, the method comprising: detecting, using animage capturing device associated with a wearable communication device,an image reflecting surface disposed in an environment of a first user,the image capturing device being forward-facing relative to the firstuser and embedded in the wearable communication device; detecting, usinga processing device, a first boundary of the image reflecting surface inresponse to the image reflecting surface being detected in theenvironment of the first user by the image capturing device; providing atactile signal of variable strength to a first side of the wearablecommunication device in response to a sensory detection signalassociated with a direction of movement of an image of the first userreflected from the image reflecting surface, the image being shiftedoutside the first boundary of the image reflecting surface, the tactilesignal guiding the direction of movement of the first user towards thefirst boundary of the image reflecting surface, the first boundarycorresponding with the first side of the wearable communication device;selecting an area of the image reflecting surface which defines an areaassociated with at least a portion of a first image within the firstboundary of the image reflecting surface; selecting, using theprocessing device, the portion of the first image displayed on the imagereflecting surface within the first boundary of the image reflectingsurface, the portion of the first image being associated with the areaof the image reflecting surface; and displaying, using a display deviceembedded in the wearable communication device, the portion of the firstimage as an overlay on a second image to a second user, the portion ofthe first image overlaying the second image, thereby being viewable bythe first user.
 2. The method, as defined by claim 1, further comprisingcontrolling selection of the portion of the first image in response toan image defining gesture associated with the first user.
 3. The method,as defined by claim 2, further comprising modifying a size of theportion of the first image in response to the image defining gestureassociated with the first user.
 4. The method, as defined by claim 3,further comprising displaying the portion of the first image as anoverlay on the second image on the image reflecting surface disposed inthe environment of the first user to be viewed by the first user.
 5. Themethod, as defined by claim 1, further comprising rotating the imagecapturing device in response to the first user.
 6. The method, asdefined by claim 1, further comprising rotating the image capturingdevice in response to a sensory image signal detected by the imagecapturing device.
 7. The method, as defined by claim 1, furthercomprising providing a tactile signal, the tactile signal causing thewearable communication device to vibrate in response to the first usermoving out of a focal point of a projected image.
 8. The method, asdefined by claim 7, further comprising generating a tactile signal tothe wearable communication device associated with the first user inresponse to a sensory detection signal associated with movement by thefirst user towards an edge of the image reflecting surface.
 9. Themethod, as defined by claim 1, further comprising generating a tactilesignal to the wearable communication device associated with the firstuser in response to a sensory detection signal associated with thedirection of movement by the first user outside a second boundary of theimage reflecting surface.
 10. A system to exchange audio-visualcommunication information between a first user of a wearablecommunication device and a second user, the system comprising: an imagecapturing device embedded in the wearable communication device, theimage capturing device being forward-facing relative to the first user;a processing device; a memory device storing instructions that, whenexecuted by the processing device, perform operations comprising:detecting an image reflecting surface disposed in an environment of afirst user; detecting a first boundary of the image reflecting surfacein response to the image reflecting surface being detected in theenvironment of the first user; providing a tactile signal of variablestrength to a first side of the wearable communication device inresponse to a sensory detection signal associated with a direction ofmovement of an image of the first user reflected from the imagereflecting surface, the image of the first user being shifted outsidethe first boundary of the image reflecting surface, the tactile signalguiding the direction of movement of the first user towards the firstboundary of the image reflecting surface, the first boundarycorresponding with the first side of the wearable communication device;selecting an area of the image reflecting surface which defines an areaassociated with at least a portion of a first image within the firstboundary of the image reflecting surface; and selecting the portion ofthe first image displayed on the image reflecting surface within thefirst boundary of the image reflecting surface, the portion of the firstimage being associated with the area of the image reflecting surface;and a display device embedded in the wearable communication device, thedisplay device displaying the portion of the first image as an overlayon a second image to a second user, the portion of the first imageoverlaying the second image, thereby being viewable by the first user.11. The system, as defined by claim 10, wherein the operations furthercomprise controlling selection of the portion of the first image inresponse to an image defining gesture associated with the first user.12. The system, as defined by claim 11, wherein the operations furthercomprise modifying a size of the portion of the first image in responseto the image defining gesture associated with the first user.
 13. Thesystem, as defined by claim 12, further comprising a display device thatdisplays the portion of the first image as an overlay on the secondimage on an image reflecting surface disposed in the environment of thefirst user to be viewed by the first user.
 14. The system, as defined byclaim 10, wherein the operations further comprise rotating the imagecapturing device in response to the first user.
 15. The system, asdefined by claim 10, wherein the operations further comprise rotatingthe image capturing device in response to a sensory image signaldetected by the image capturing device.
 16. The system, as defined byclaim 10, wherein the operations further comprise providing a tactilesignal, the tactile signal causing the wearable communication device tovibrate in response to the first user moving out of a focal point of aprojected image.
 17. The system, as defined by claim 16, wherein theoperations further comprise generating a tactile signal to the wearablecommunication device associated with the first user in response to asensory detection signal associated with movement by the first usertowards an edge of the image reflecting surface.
 18. The system, asdefined by claim 10, wherein the operations further comprise generatinga tactile signal to the wearable communication device associated withthe first user in response to a sensory detection signal associated withthe direction of movement by the first user outside a second boundary ofthe image reflecting surface.
 19. A computer-readable device storinginstructions that, when executed by a processing device, performoperations comprising: detecting an image reflecting surface disposed inan environment of a first user using an image capturing device, theimage capturing device being forward-facing relative to the first userand embedded in a wearable communication device; detecting a firstboundary of the image reflecting surface in response to the imagereflecting surface being detected in the environment of the first user;providing a tactile signal of variable strength to a first side of thewearable communication device in response to a sensory detection signalassociated with a direction of movement of an image of the first userreflected from the image reflecting surface, the image of the first userbeing shifted outside the first boundary of the image reflectingsurface, the tactile signal guiding the direction of movement of thefirst user towards the first boundary of the image reflecting surface,the first boundary corresponding with the first side of the wearablecommunication device; selecting an area of the image reflecting surfacewhich defines an area associated with at least a portion of a firstimage within the first boundary of the image reflecting surface;selecting the portion of the first image displayed on the imagereflecting surface within the first boundary of the image reflectingsurface, the portion of the first image being associated with the areaof the image reflecting surface; and displaying the portion of the firstimage as an overlay on a second image to a second user, the portion ofthe first image overlaying the second image, thereby being viewable bythe first user.
 20. The computer-readable device of claim 19, whereinthe operations further comprise: controlling selection of the portion ofthe first image in response to an image defining gesture associated withthe first user; modifying a size of the portion of the first image inresponse to the image defining gesture associated with the first user;and displaying the portion of the first image as an overlay on thesecond image on the image reflecting surface disposed in the environmentof the first user.